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INTRODUCTION 


Using a color- television display system and interactive 
graphics equipment on-line to an IBM 360/44 computer, investigators 
at the Manned Spacecraft Center have developed a variety of inter- 
active displays which aid in analyzing remote sensor data. This 
paper describes how such interactive displays are used to: 

1) Analyze data from a multispectral scanner. 

2) Develop automatic pattern recognition systems based on 
multispectral scanner measurements. 

3) Analyze data from non- imaging sensors such as the infrared 
radiometer and microwave scatterometer . 


DESCRIPTION OF DISPLAY/GRAPHICS SYSTEM 


The work described in this report was performed in the 
computer laboratory of the Manned Spacecraft Center Information 
System Division. This laboratory includes a Digital Television 
Display System (DTDS) and a Grafacon tablet on-line to an IBM 360/44 
digital computer. 
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HARDWARE 


The DTDS consists of eight channels of recirculating memory 
each capable of storing a complete 729-line television frame. The 
system hardware includes a symbol library which makes it possible 
to display the various alphanumeric characters at any specified 
location within the frame as in Figures 9-13. 

The intensity at each raster point within the frame is encoded 
in the recirculating memory using a single binary bit; i.e., each 
raster point is either on with full intensity or completely off. 

In order to obtain a gray scale the frame is divided into cells 
each of which can have a different number of raster points inten- 
sified. For example, in Figure 2 the intensity variation (in each 
color) was obtained by dividing the frame into 160 x 111 cells each 
having 5x5 raster points. The intensity of each cell is set 
by controlling the number (0 to 25) of intensified raster points. 
The various 5x5 dot matrices are stored as special symbols in the 
hardware symbol library. 

Another important component .of the DTDS hardware is the vector 
generator. This unit takes as its input the display coordinates of 
two points and automatically generates the display coordinates of 
all points on the straight line connecting them. This feature is 
used in forming the displays shown in Figures 17 - 19. 

The display device used for all of the color photographs 
presented in this paper is the precision color monitor shown 
schematically in Figure 1. It consists of three cathode ray tubes 
having red, green and blue phosphors; each tube (i.e., color) is 
intensified by a separate channel of recirculating memory. The 
images on the three tubes are superimposed and projected onto a 
ground-glass viewing screen by a combination of mirrors, dichroic 
filters and a projection lens. Accurate registration of the three 
images is maintained by various electronic (e.g., deflection gain 
and centering) and mechanical (e.g., mirror alignment) adjustments. 

Another output device connected to the DTDS is the Xerox hard- 
copy machine. It is used for copying black-and-white alphanumeric 
and graphic displays such as Figures 9, 10 and 19. 

The user communicates graphical and alphanumeric information 
to the computer by way of a Grafacon tablet and a keypack, 
respectively. The Grafacon tablet is a device which 30 times per 
second transmits to the computer the instantaneous coordinates of 
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a pen held by the investigator. The tablet is 10 in. on a side and 
the coordinates are quantized to 0.01 in. As explained later, the 
operator uses the Grafacon pen to define certain areas and intervals 
of interest; for example, see Figures 3, 4, 17 and 18. The keypack, 
similar to a typewriter keyboard, contains all of the alphanumeric 
symbols. It is used to assign numbers to the areas and intervals 
of interest in Figures 3, 4, 17 and 18. 


SOFTWARE 


All of the programs used to generate the displays reported in 
this paper were written in FORTRAN IV. This was possible because 
the DTDS system is supported by an extensive software package 
consisting of many FORTRAN-callable assembler language subroutines 
(Reference 1) . 

DISPLAYS USED TO ANALYZE MULTISPECTRAL SCANNER DATA 


INTRODUCTION 


The multispectral scanner collects radiant energy from a small 
area (i.e., resolution cell) on the ground and separates it into 
spectral components in each of twelve wavelength bands between 
0.40y and l.OOy . The instantaneous output signals are recorded 
and subsequently converted to numbers in the range 0-255 by an 
analog- to-digital conversion process.* The instantaneous field- 
of-view is scanned rapidly across the flight- line by a rotating 
mirror; the data is sampled at 222 points within each scan line. 
Successive scan lines are displaced along the flight line by the 
forward motion of the aircraft. The result is that multispectral 
scanner measurements are specified (i.e., referenced) on the basis 
of three independent variables: 1) scan line, 2) sample within a 

scan line, and 3) channel number. 


*The multispectral scanner data presented in this paper was pro- 
cessed by the Laboratory for Applications of Remote Sensing 
(LARS) at Purdue University. In addition to the multispectral 
scanner data, LARS also provided the ground truth information 
necessary to conduct pattern recognition studies. 
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COLOR DISPLAY OF MULTISPECTRAL SCANNER DATA 


Figure 2 is a photograph taken from the Digital Television 
Display System color monitor showing an approximately natural- 
color picture of the ground scene as reconstructed from the 
multispectral scanner data. This display consists of Lines 1 to 
160 in the horizontal direction and Samples 1 to 111 in the vertical 
direction; this represents an area 2/3 x 1/2 mi. on Purdue Flight 
Line C-l flown on June 26, 1966. 

At each display cell the intensity of the blue, green, and 
red is set according to the multispectral scanner outputs at 
that particular Line and Sample in Channels 2, 6 and 10 (i.e., 
radiance measured at that particular ground cell in the spectral 
ranges 0.44-0.46y , 0.52-0.55P , and 0.66-0.72y , respectively). 

For each of the three different colors the user can specify the 
intensity (one of 16 levels) associated with any multispectral 
scanner output (in the range 0-255) . The user can also associate 
any one of the twelve multispectral scanner channels with each of 
the three colors; in this way a variety of "false-color" displays 
can be generated. 

The necessary annotation data is included as an integral part 
of the display. For example, the following information is recorded 
in the top line of Figure 2: 

1) The blue, green, and red intensities depend on Channels 2, 
6, and 10, respectively. 

2) The data is from Purdue Flight Line C-l flown on 26 June 
1966. 

3) The display consists of Lines 1-160 and Samples 1-111 in 
steps one.* 


INTERACTIVE METHOD FOR ENTERING GROUND- TRUTH INFORMATION 


One of the most important functions which an investigator 


"The program also has a provision for displaying every other Line 
and/or every other Sample. 
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must perform in analyzing remote sensor data is to put ground- 
truth information into the computer. The Digital Television • 

Display System is well-suited to this requirement in that it 

1) displays a complete frame rather than a single scan line at 
a time, 2) is available immediately rather than requiring film 
processing, and 3) is responsive to on-line keyboard and graphical 
input devices. 

In processing multispectra 1 scanner data it is necessary to 
put into the computer certain training and/or test fields (Reference 

2) . These are areas where the material classification (e.g., crop- 
type) is known from a ground- truth survey. In the interactive system 
developed at the Manned Spacecraft Center the investigator uses a 
Grafacon pen to outline fields he recognizes in the natural or 
false-color display shown in Figure 2. 

Movement of the Grafacon pen causes a corresponding line to be 
displayed on the monitor. Using this display, the investigator 
moves the pen so as to outline the field of interest.* Once the 
boundary has been closed the computer responds by erasing the 
enclosed field from the display. The investigator then uses the 
keypack to type in the field number designation for this area. 

The investigator is assisted in the interactive dialog with 
the system by computer- genera ted instructions displayed in the upper- 
left corner of the frame (e.g., see Figure 3). The table on the 
next page gives the sequence of instructions and the investigator's 
response. A short 16 mm color movie (NASA/MSC # S-70-370 available 
from the authors) shows the procedure used to set into the computer 
the boundaries of several different fields. 

Figure 4 shows the result of outlining all fields within the 
frame. Here all areas (except roads) have been erased from the 
display leaving only the boundaries and associated field numbers. 

This information is also printed out in the form of a Xerox hard- 
copy for permanent documentation. 


*It should be emp la sized that the fields can have any shape and 
orientation; for example, see Figure 3. 
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Computer- Genera ted Instruction 

Investigator's Response 

OUTLINE A FIELD 

Use the Grafacon pen to outline 
the area of interest. 

DRAW BOUNDARY AGAIN 

Outline the area again, this time 
making sure to close the 
boundary . 

POSITION FIELD NUMBER 

Use the Grafacon pen to point to 
the place in the erased area 
where the field number should be 
displayed . 

TYPE FIELD NUMBER 

Use the keypack to type in the 
number associated with the out- 


lined area. 


GROUND- TRUTH MAP 


The boundary information shown in Figure 4 is recorded on 
magnetic tape and used in subsequent processing to derive a 
ground-truth map in computer-readable form. This is a magnetic 
tape giving the material-type at every point (i.e., Line-Sample 
combination) within the outlined areas. 

To obtain the ground- truth map the investigator only needs to 
specify (in the form of data cards) the material-type to be 
associated with each field number.* Using an algorithm described in 
Reference 3, the computer automatically "fills in" the outlined 
fields with the designated material-types and records the result on 
magnetic tape. In this way a ground- truth tape was prepared for the 
entire flight line and is available from the authors. 


*In the case of the data presented here (Purdue Flight Line C-l, 
June 26, 1966) the necessary information resulted from a ground- 
truth survey conducted by the Laboratory for Applications of 
Remote Sensing. 
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Figure 5 is a color coded ground- truth map formed by displaying 
Lines 1 to 160 and Samples 1 to 111 of the ground- truth tape. In 
this figure the colors red, green, yellow, and blue represent areas 
of soybean, wheat, oats, and bare soil, respectively. The white 
areas are fields where no crop- type designation was available. 


COMPUTER DERIVED LAND-USE MAP 


Using algorithms described in the next section the computer 
processes the multispectral scanner data to determine automatically 
what material is present within each resolution element. This 
determination is based on data from several (e.g., four) channels at 
any given point. 

The computer-derived results are displayed in the form of a 
color coded land-use map such as Figure 6. In this display the 
color assigned to a given point (i.e., Line-Sample combination) 
depends on the crop- type derived by the computer. As in Figure 5 
the colors red, green, yellow and blue represent areas of soybean, 
wheat, oats, and bare soil, respectively. 


ERROR MAP 


By comparing Figures 5 and 6 it can be seen that the computer 
derived results generally agree with the actual ground-truth. To 
facilitate this comparison an error map display was developed. This 
is a presentation showing the correct crop- type (as its own distinc- 
tive color) at every point where the computer derived classification 
is incorrect; areas correctly classified (and white areas where no 
crop- type designation was available) are left blank. Figure 7 is 
an error map which shows the differences between Figures 5 and 6. 


DISPLAYS USED FOR PATTERN RECOGNITION RESEARCH 


INTRODUCTION 


The term "pattern recognition" is often applied to the various 
computer-based methods by which material- type classifications are 
derived automatically from multispectral scanner measurements. The 
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pattern recognition process can be explained by reference to 
Figure 8. Here a point in the three-dimensional vector space repre- 
sents the tip of a measurement vector having as its components the 
measured outputs from three spectral bands at a single resolution 
cell on the ground. The pattern recognition process is based on the 
fact that measurement vectors for different materials tend to cluster 
in different regions of measurement space. Pattern recognition 
systems operate in two stages: 

1) The multidimensional measurement space is partitioned into 
non- over lapping regions associated with each type of 
material. The regions may be described analytically 
(Reference 2) or prestored in computer memory (Reference 4). 

2) The measurement vector for each resolution cell is analyzed 
to determine which region it occupies in measurement space. 

The displays shown in Figures 9-16 were developed to aid in the 
design of such pattern recognition systems. 


MULTIDIMENSIONAL HISTOGRAM 


In order to observe the clustering of measurements from a 
given material it is helpful to plot in three-dimensional space the 
measurement vectors of all resolution cells inside the training 
fields for that material; this is shown in Figure 8. It is possible 
to present this same information in two dimensions by plotting only 
those points which fall within a specified slice (i.e., range of 
values) in the third dimension. For example, in Figure 8 a training 
sample having the value x in Channel X, the value y in Channel Y, and 
a value in the range z^ to Z 2 in Channel Z is plotted at the point 
(x, y) . A multidimensional histogram for a given material is 
formed by counting the number of training samples at the point 
(x, y) and in the range z-^ to Z£. 

Figure 9 is a multidimensional histogram for soybean. This 
type of display is a quantitative version of the "Color-Space" plot 
described by Holter (Reference 5). At any point in the Channel 1 - 
Channel 9 plane is given the number of soybean training samples 
which are in the range 151 to 158 for Channel 11 and the range 179 
to 186 for Channel 12. The material- type and the Channel 11 - 
Channel 12 slice are specified at the top of the display. The 
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range* of values in Channel 1 and in Channel 9 are displayed on the 
horizontal and vertical axes, respectively. 

Figure 10 is a histogram for corn plotted over the same range 
in Channels 1 and 9 and for the same slice in Channels 11 and 12. 
From Figures 9 and 10 it can be seen** that multispectral scanner 
measurements from corn lie above and to the right of measurements 
from soybean. The computer-based pattern recognition system to be 
described next uses this as a basis for discriminating between 
soybean and corn. 


CLASSIFICATION SPACE 


As explained previously, pattern recognition involves 
partitioning the measurement space into non-overlapping regions 
associated with each material-type. A common analytic method for 
deriving the classification space assumes a Gaussian probability 
density function and employs a maximum- likelihood decision rule. 


Equation (1) gives the probability that Material i gave rise 
to the given measurement vector X. 


P.(X)= 
where : 


* exp[ — 1/ 2 ( X - M.) T K." 1 (X - M.)] Eq . (1) 


1- ' (2tt) W/2 |K. 


Pi=Probability density function for the i^ material 
X =Measurement vector for a particular ground cell 
N =Number of channels used 
Mj_=Mean vector for the i 1 ^ material 
K^=Covariance matrix for the i 1 " material 


*The display provides only 16 points along the horizontal and verti- 
cal axes; however, each point can represent any integer number of 
multispectral scanner units. For example, in Figures 9 and 10 each 
point represents a cell 3x3 multispectral scanner units in 
the Channel 1 - Channel 9 plane. 

**The arrows in Figures 9 and 10 point to the same position in the 
Channel 1 - Channel 9 plane. These arrows were inserted for the 
convenience of the reader and are not part of the computer- 
generated display. 
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The mean vector I'L and the covariance matrix which appear as 
parameters in Equation (1) are evaluated from the training samples 
of Material i. These are measurements from resolution cells inside 
training fields similar to the ones outlined in Figure 4. 


The maximum- likelihood decision rule classifies a given sample 
as Material i if the probability of Material i computed from 
Equation (1) is higher than the probability of any other material. 
However, it is sometimes preferable to assign a sample to the 
category "Unknown" rather than misclassify it. This is accomplished 
by the parameters L and D in the decision rule given by Equation (2). 

CLASSIFY THE GIVEN GROUND CELL AS MATERIAL i IF: 


Pi (X) £ L-P.(X) and 

1/2 (X-Mi) 1 K^ 1 (X-M i )^ D 


where : 


L = Likelihood ratio 
D = Threshold distance 

Here a sample is classified as Material i only if the following 
two conditions are met: 

(1) The probability of Material i must be at least L times 
greater than the probability of the second most likely 
material. 

(2) The normalized distance between the measurement vector and 
the mean vector for Material i must be less than D. 

Displays were developed to study the effect of L and D on the 
classification space. 

Figures 11 - 16 are classification space displays. For specific 
values in Channels 11 and 12 they show at each point in the Channel 
1 - Channel 9 plane the material- type derived from Equations (1) 
and (2). To facilitate comparison with the multidimensional 
histograms the range of values in Channel 1 and in Channel 9 are 
the same as for Figures 9 and 10. Also the specific values for 
Channels 11 and 12 (i.e., 154 and 182, respectively) are in the 
middle of the Channel 11 - Channel 12 slices. 

Figure 11 shows the classification space for the case L = 1 
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and D = 1000; this corresponds to the maximum- likelihood decision 
rule. In this display the letters S, C, 0, R, W, and Y represent 
soybean, corn, oats, red clover, wheat and rye, respectively. 

Figure 12 shows the case L = 5 and D = 1000. Here no classification 
is made unless the probability of the most likely material is at 
least five times the probability of the second most likely material. 
The result is that regions on the boundaries between different 
materials are classified as "Unknown" (i.e., no classification is 
made). Figure 13 shows the case L = 1 and D = 15. Here no classi- 
fication is made if the measurement vector is further than the 
specified distance from the mean vector. In four-dimensional 
measurement space a threshold setting of D=15 excludes the correct 
classification less than 1% of the time. 

In Figure 14 the region of measurement space associated with 
soybean is represented by the yellow area* for the case L = 1 and 
D = 10. The effect of increasing the threshold distance to D = 20 
is to expand the soybean region to include the red area below and to 
the left. The reason the expansion is not symmetrical will be 
discussed in connection with Figure 16. 

Figure 15 is a composite display formed by superimposing the 
multidimensional histogram for soybean (see Figure 9) on the soybean 
classification region. It can be seen that almost all the samples 
are correctly classified because they fall within the soybean 
region. Shown in red are the number of soybean samples which would 
be incorrectly classified as some other crop- type or classified as 
"Unknown" . 

Figure 16 is a composite display formed by superimposing the 
multidimensional histogram for corn (see Figure 10) on the soybean 
classification region. Shown in yellow are the number of corn 
samples which would be incorrectly classified as soybean. From 
this figure it can be seen that expansion of the soybean classifi- 
cation region upward and to the right would result in many corn 
samples being misclassified as soybean. This is the reason for the 
asymmetrical expansion of the soybean region in Figure 14 when the 
threshold distance was increased from 10 to 20. 


*This area is equivalent to the area cut out from the classification 
mask in the "light-pencil" technique for pattern recognition 
developed at the University of Michigan (Reference 5). 
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DISPLAYS USED TO ANALYZE NON- IMAGING SENSOR DATA 


INTRODUCTION 


Unlike the multispectral scanner, many sensors produce output 
signals which cannot be displayed in the form of imagery such as 
Figure 2. This limitation results from the fact that they a) have 
very poor resolution and/or b) provide only a line scan over the 
ground scene. Non- imaging sensors currently used in the Earth 
Resources program include the infrared radiometer, infrared 
spectrometer, microwave radiometer and microwave scatterometer . 

In order to analyze data from these non- imaging sensors it is 
helpful to display their output signals in spatial registration 
with ancillary imagery of the ground scene. References 3 and 6 
describe a method for producing computer- genera ted transparencies 
which are superimposed over aerial photographs to show the exact 
sensor boresight path and the resulting output signals. Using these 
overlays an analyst can a) determine the sensor response for 
various materials which can be recognized from the photograph and/or 
b) identify the material which gave rise to any given sensor 
response . 

This overlay approach has been extended to permit on-line 
interactive analysis using the Digital Television Display System. 

The black and white aerial photograph (e.g., a 9 x 9 in. RC8 
camera frame) is scanned by a closed-circuit television camera and 
the resulting video signal is presented on the green tube of the 
precision color monitor. The computer-generated graphics showing 
the sensor boresight path and the output signals are displayed on 
the red tube. The two displays are superimposed* and projected onto 
the ground-glass viewing screen as shown in Figure 1. Using the 
Grafacon pen and the keypack on-line to the computer the analyst 
can define flight path intervals over which the computer processes 
the sensor data. This approach will be explained by reference to 
infrared radiometer and microwave scatterometer data. 


*The magnification and alignment of the closed-circuit television 
camera are carefully maintained so that the resulting image on the 
monitor is in exact spatial registration with the computer- 
generated graphics. 
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INFRARED RADIOMETER 


The green part of Figure 17 is the closed-circuit television 
presentation of RC8 Frame 4419 taken during Mission 102. It shows 
an island (actually a spoil sand bank) adjacent to the intracoastal 
waterway south of Corpus Christi, Texas; the surrounding dark areas 
are cloud shadows. The red part of Figure 17 is computer- generated 
graphics showing the sensor bores ight path* and resulting output 
signal from the infrared radiometer. From this display it is 
apparent that the path of the sensor passed directly over the island 
and that the island was significantly warmer than the surrounding 
water . 

The analyst interacts with the display by defining certain 
intervals along the sensor boresight path. He does this by using 
the Grafacon pen to point to the beginning and ending points of each 
interval. The computer responds by displaying a short mark across 
the sensor path at the selected position; six such points define 
three different intervals in Figure 17. After the analyst signals 
that he has set in all intervals of interest the system responds 
by computing the average sensor signal over each interval. The 
three average temperatures are displayed across from their corre- 
sponding intervals in Figure 17. 


MICROWAVE SCATTEROMETER 


Figure 18 shows the boresight paths and output signals for four 
different look-angles of the microwave scatterometer . The look- 
angles (displayed in the upper- left corner) are 2, 5, 10, and 20 
degrees forward from nadir. Sigma Zero, the reflectance coefficient 
for each look-angle, is plotted in spatial registration with the 
ground scene. It is apparent from this display that the reflectance 
coefficient increases as the beam passes over the island. 

As before, the analyst interacts with the display by using the 
Grafacon pen to define intervals along the sensor path. In the case 
of the microwave scatterometer, however, the computer responds by 


*References 3 and 6 describe in detail how the sensor boresight path 
(relative to aerial photographs) is computed using aircraft 
guidance and navigation data recorded in flight. 
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displaying the average value of reflectance coefficient for all four 
look-angles in each interval. For example, in Interval #1 the 
average reflectance coefficients are -6, -16, -33, and -37 decibels 
for look-angles of 2, 5, 10, and 20 degrees, respectively. 

The system also produces a Xerox hard-copy showing a graph of 
the reflectance coefficient versus look-angle for each interval. 
Figure 19 gives the microwave scatterometer signatures for the three 
intervals defined in Figure 18. Such displays can be used to deter- 
mine whether the signatures are different enough to permit discrim- 
ination between materials of interest. 


CONCLUSIONS 


An interactive display/graphics system has been developed for 
remote sensor data analysis. This system has the following advan- 
tages over other systems being used currently: 

1) It displays digital data in color a full frame at a time. 

2) The display is available immediately because no photo- 
graphic processing is involved. 

3) It permits the analyst to interact with the display and the 
data processing. 

The display system has been used in the following applications: 

1) Natural and false color displays of multispectral scanner 
data. 

2) Ground- truth and land-use maps. 

3) Multidimensional histograms and classification-space plots 
for pattern recognition research. 

4) Displays showing non- imaging sensor data in spatial 
registration with ancillary imagery. 

5) Microwave scatterometer signatures. 

The interactive graphics capability has been used for the following 
purposes: 

1) To define training and test fields by setting into the 
computer boundaries and ground- truth information. 

2) To define intervals along the boresight paths of non- 
imaging sensors. 

The development of such systems has considerable programmatic 
importance in that they make it possible to analyze large quantities 
of remote sensor data in a short time. 
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Figure 1. -Schematic diagram of precision 
color monitor. 
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Figure 2.- Natural-color display 
of multispectral scanner data. 


Figure 3.- Interactive method for 
entering ground-truth areas. 




Figure 4.- Completed boundaries 
for ground- truth areas. 


Figure 5.- Display showing color 
coded ground-truth map. 




Figure 6.- Display showing com- 
puter derived land-use map. 


Figure J.~ Errors in the computer 
derived land-use map. 
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Figure 8.- Multispectra 1 scanner data 
represented as points in three- 
dimensional measurement space.' 
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Figure 9.- Multidimensional histogram 
for soybean. 


Figure 10.- Multidimensional histogram 
for corn. 
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Figure 11.- Classification space Figure lk.- Areas classified as 

derived using L=1.0 and D=1000. soybean for D=10 and D=20. 



Figure 12.- Classification space 
derived using L=5-0 and D=1000. 



Figure 15.- Soybean histogram on 
area classified as soybean. 



Figure 13.- Classification space Figure 16.- Corn histogram on area 

derived using L=1.0 and D=15. classified as soybean. 
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Figure 17.-’ Bores ight path and 
sensor return for infrared 
radiometer. 
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Figure 18.- Boresight path and 
sensor return for microwave 
scatterometer . 
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Figure 19 •- Microwave scatterometer signatures for 
three different flight-path intervals. 


582 






